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ABSTRACT 

We consider joint transceiver design for general Multiple-Input 
Multiple-Output communication systems that implement interfer- 
ence (pre-)subtraction, such as those based on Decision Feedback 
Equalization (DFE) or Tomlinson-Harashima precoding (THP). We 
develop a unified framework for joint transceiver design by consider- 
ing design criteria that are expressed as functions of the Mean Square 
Error (MSE) of the individual data streams. By deriving two in- 
equalities that involve the logarithms of the individual MSEs, we 
obtain optimal designs for two classes of communication objectives, 
namely those that are Schur-convex and Schur-concave functions of 
these logarithms. For Schur-convex objectives, the optimal design 
results in data streams with equal MSEs. This design simultaneously 
minimizes the total MSE and maximizes the mutual information for 
the DFE-based model. For Schur-concave objectives, the optimal 
DFE design results in linear equalization and the optimal THP de- 
sign results in linear precoding. The proposed framework embraces 
a wide range of design objectives and can be regarded as a counter- 
part of the existing framework of linear transceiver design. 

Index Terms — Decision Feedback Equalization, Tomlinson- 
Harashima precoding, transceiver design, MIMO channels. 

1. INTRODUCTION 

One of the key advantages of Multiple-Input Multiple-Output 
(MIMO) communications schemes is that they facilitate the simul- 
taneous transmission of multiple data streams. Typically, such 
schemes involve processing of the data streams at the transmitter 
(precoding) and processing of the received signals (equalization) to 
"match" the transmission to the channel and to mitigate the inter- 
ference between the received streams at reasonable computational 
cost. One approach to the design of such a scheme is to focus on 
linear precoding and linear equalization; e.g. [1, 2]. An alterna- 
tive approach that offers some advantages is to allow interference 
(pre-)subtraction at either the transmitter or the receiver. This ap- 
proach includes schemes with linear precoding and Decision Feed- 
back Equalization (DFE), and schemes with Tomlinson-Harashima 
precoding (THP) and linear equalization, and will be the focus of 
this paper. 

A large number of design strategies have been proposed for the 
class of linear MIMO transceivers (e.g., [2]), and a uniform frame- 
work that encompasses many of these designs was proposed in [1]. 
This framework consists of functions that capture a broad range of 
communication objectives, namely those that are Schur-convex and 
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Transmitter DFE Receiver 

Fig. 1. MIMO transceiver with Decision Feedback Equalization. 

Schur-concave functions of the mean square error (MSE) of each 
data stream. For the class of interference (pre-)subtraction, designs 
for DFE based schemes using an MMSE criterion receiver were con- 
sidered in [3, 4], and designs subject to a zero-forcing constraint 
were considered in [5, 6]. Some THP counterparts of these designs 
were presented in [3] and [7], respectively. 

In this paper, we develop a broadly applicable framework for 
joint transmitter and receiver design for MIMO systems with a DFE 
or a THP. We consider the broad range of design criteria that can be 
expressed as either Schur-convex or Schur-concave functions of the 
logarithm of the MSE of each data stream, and we provide optimal 
transceiver designs for these two classes. In addition to providing a 
generalization of existing designs based on the overall MSE, these 
classes of functions embrace other design criteria such as minimiz- 
ing the maximum of the individual MSEs, or minimizing a weighted 
geometric mean of the MSEs. Moreover, for the DFE model, design 
criteria expressed in terms of the signal to interference-plus-noise ra- 
tio (SINR) and bit error rate (BER) of each stream are included in the 
set of objectives covered by these classes. Interestingly, the optimal 
design for both Schur-convex and Schur-convex objectives yields a 
diagonal MSE matrix. For Schur-convex objectives, the optimal de- 
sign results in data streams with equal MSEs. Furthermore, for the 
DFE model, the optimal design for this class simultaneously min- 
imizes the total MSE and maximizes the mutual information. For 
Schur-concave objectives, the optimal design results in linear pre- 
coding and equalization. From a boarder prospective, the proposed 
framework can be viewed as a counterpart for the design of DFE- 
based and THP-based transceivers of the unified framework for the 
design of linear transceivers in [1]. 

2. TWO SYSTEM MODELS 

We consider a generic MIMO communication system in which the 
received signal can be written as y = Hx + n, where H £ C^*" x * 
represents the channel, the transmitted vector x is synthesized from 
a vector s 6 C of data symbols, and the additive noise has zero- 
mean and covariance matrix E„{nn ff } = R„. We will consider 
a general design approach that encompasses several design criteria 
for two communication systems, namely those systems with linear 
precoding at the transmitter and a DFE at the receiver, and those 
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Fig. 2. MIMO transceiver with Tomlinson-Harashima precoding 



For moderate to large values of M this power increase is negligible 
and the approximation E{vv ff } = I can be used. We will use the 
more accurate approximation Ejvv"} = a%\; e.g., [3, 7]. 

For the THP scheme, the received signal vector can be written as 
y = HPC x u + n, and hence the receiver's estimate of the of the 
modified data symbols is u = G fl HPC^u + G H rs.. Following 
linear equalization, the modulo operation is used to eliminate the 
effect of the periodic extension of the constellation induced at the 
transmitter. In terms of the modified data symbols, the error signal 
e = u u = G H HPv + G H w — Cv can be used to define the 
Mean Square Error matrix E = E v {ee ff }: 
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Fig. 3. Equivalent linear transmitter model for THP-based system 



E = alCC H - aJCP ff H H G - <j 2 G h HPC h 

+ ffjG H HPP ff H fl G + G H R„G. (2) 

For the TH precoding model, the transmitter power constraint is 
given by E{xx w } = <r>(PP H ) < P tota . 



systems with THP at the transmitter and linear equalization at the 
receiver. (The linear transceiver is a special case of both systems 
with the feedback matrix B = 0; see Figs 1 and 2.) 

2.1. Decision Feedback Equalization 

As shown in Fig. 1 , the transmitted vector is generated by linear pre- 
coding, x = Ps, and hence the received vector y = HPs + n. 
The DFE is implemented using a feedforward matrix G H and a 
strictly lower triangular feedback matrix B G C. Assuming cor- 
rect previous decisions, the vector of inputs to the quantizer is 
s = (G ff HP - B)s + G H n. Defining the error signal e = s - s, 
and using the assumption E s {ss H } = I, the mean square error ma- 
trix can be written as: 

E = E s {ee H } = CC H CP H H H G - G H HPC H 

+ G H UPP H H H G + G H R. n G, (1) 

where C = I + B is a unit diagonal lower triangular matrix. The ob- 
jective is to design the G, C, P for different design criteria, subject 
to the transmitter power constraint E s {xx H } = tr(PP H ) < P to tai- 

2.2. Tomlinson-Harashima Precoding 

As shown in Fig. 2, in THP the transmitter performs successive in- 
terference pre-subtraction and spatial precoding using the strictly 
lower triangular matrix B and the precoding matrix P, respectively. 
We assume that the elements of s are chosen from a square QAM 
constellation 5 with cardinality M and that Esjss^} = I. The 
Voronoi region of this constellation, V, is a square whose side length 
is D. Following pre-subtraction of the effect of previously precoded 
symbols, the transmitter uses the modulo operation so that the sym- 
bols of v lie within the boundaries of V. The effect of the mod- 
ulo operation is equivalent to the addition of ij. = ¥ k e D + i* ma9 £) 
to Sfe, where i£ e , i™" 9 g Z. Using this observation, we obtain 
the standard linearized model of the transmitter shown in Fig. 3 
(e.g. [7]), in which v = (I + B) _1 u = C _1 u. As a result 
of the modulo operation, the elements of v are almost uncorre- 
cted and uniformly distributed over the Voronoi region V [7, Th. 
3.1]. Therefore, the symbols of v will have slightly higher aver- 
age energy than the input symbols s. For a square QAM, we have 
a 2 = E{jv fc | 2 } = I J^ T E{|s fc | 2 } for all k except the first one [7]. 



2.3. General Model 

From equations (1) and (2), we observe that the MSE matrix E of 
both systems has a common form: 

E = a 2 CC H -a 2 CP H H H G a 2 G H UPC H + G H R V G, (3) 

where R H = cr 2 HPP H H H + R„. For the DFE model a 2 = 1 
while for the TH precoding model a 2 = a 2 . The average transmitter 
power constraint can be rewritten as ti(PP H ) < Ptotai/o" 2 = P. 



3. OPTIMAL FEEDFORWARD AND FEEDBACK 
MATRICES 

We consider the joint design of the transceiver matrices G, C, P in 
order to optimize system design criteria that are expressed as func- 
tions of the MSE of the individual data streams En. We will adopt 
three-step design approach. First, an expression for the optimal feed- 
forward matrix G H will be found as a function of C and P. Second, 
using the expression of the optimal G, an expression of the optimal 
C will be found as a function of P. Finally, using the obtained ex- 
pressions of G and C, we will design the optimal precoder P. 



3.1. Optimal feedforward matrix G 

For given C and P, the MSE of the i th data stream, En, is a convex 
function of the i th column of G, denoted g 4 , and is independent of 
other columns. Therefore, the columns of G can be independently 
optimized to minimize the individual MSEs. A similar property was 
observed in [1] for linear transceivers. Setting the gradient of En 
with respect to gi to zero, we obtain following expression for the 
optimal G: 

G = a 2 R ! ; 1 HPC ff . (4) 

Since each g; independently minimizes the MSE of the i th data 
stream, the expression of G in (4) is also the optimal feedforward 
matrix in the sense of the sum of MSEs, tr(E). Using this expres- 
sion, the MSE matrix can be written as: 

E = ct 2 C(I + ff 2 P H H fi R; 1 HP)" 1 C fi = CMC", (5) 

where the matrix inversion lemma has been used. 



3.2. Optimal feedback matrix B 

From (5) we observe that the MSE of each data stream En is convex 
function of the i lb row of C = I + B and is independent of the 
other rows. Therefore, the optimal C that minimizes the individual 
MSEs can be obtained by minimizing any convex combination of 
Ejf. By choosing that convex combination to be the sum, our goal 
reduces to minimizing tr(CMC H ) subject to C being unit diagonal 
lower triangular matrix. Using the Cholesky decomposition M = 
LL W , where L is a lower triangular matrix with positive diagonal 
elements, we can rewrite the objective as tr(CMC H ) = |CL||f-, 
where || ■ \\p denotes the Frobenius norm and the product CL is 
positive definite lower triangular matrix [8]. Let Ai(CL) > . . . , > 
Ax(CL) and cti(CL) > . . . > <tk (CL) denote the ordered eigen 
values and singular values, respectively, of the matrix CL. Then the 
unit diagonal lower triangular C that minimizes tr(CMC H ) can be 
obtained using the following lower bound: 

K 

l|CL|^ = Eti^(CL) > E A ?( CL ) 

i=l 

K K 

= ^(CL)^^4 (7) 

i=l i=l 

where the bound in (6) is obtained by applying Weyl's inequality [9], 
and (7) follows from the fact that CL is lower triangular and C is 
unit diagonal. The inequality in (6) is satisfied with equality when 
the matrix is normal [9]. Since our matrix CL is a triangular matrix, 
it can only be normal if it is diagonal [8, pp 103]. Therefore, the 
matrix C that attains the lower bound is: 

C = Diag(L 11 ,...,LKx)L- 1 . (8) 

Using this optimal C, the MSE matrix can be rewritten as: 

E = Diag (L? L 2 KK ). (9) 

We observe that for any given precoding matrix P, the optimal feed- 
forward and feedback matrices will yield a diagonal MSE matrix, 
with the individual MSEs being En = Y? u . 

4. OPTIMAL PRECODING MATRIX P 

Given the optimal G and C, the last step is to design a precoding 
matrix P to optimize design criteria expressed as functions of indi- 
vidual MSE of each stream, L%. We will first derive two inequalities 
involving La that enable us to characterize the optimal precoder. 

4.1. Preliminaries 

To derive the first inequality, we will use the concept of multiplica- 
tive majorization: 

Multiplicative Majorization [9, 10]: Let a, b £ and let a^j > 
• • • > a [K] denote the elements of a in descending order. The vector 
b is said to multiplicatively majorize a, a ^ x b, if Yll=i a [»] < 
UUi b[i] , for j = 1, . . . , K - 1 and ]jf =1 a w = TjJLi b w . 
An important example of this definition is: 

Lemma 1 Weyl [9]: Let A G C KxK and let A, (A) and a; (A) 
denote the eigen values and singular values of A, respectively. Then 
we have (IMA)^, |Ak(A)| 2 )^ x (a 2 (A), a 2 K (A)). 

If A is normal, then |A;(A)| = ai(A). 



Applying the above lemma to the positive definite lower triangular 
matrix L, we obtain out first inequality: 

(L?i, . • • , 1>kk) (<r?(L),...,<&(L)). (10) 

The second inequality involves the more common notation of 
additive majorization: 

Additive Majorization [ 10]: Let a, b £ R K . The vector b is said to 
majorize a, a -< b, if J2i=i a [i] < ELi b M > for 3 = 1, • ■ • , K ~ 
1 and a M = x b H 

We observe that if elements of a and b are positive, then a -< x 
b <=> ln(a) -< ln(b). Consequently, (10) can be written as: 

l<m, (11) 

where I = (lnLfi, . . . , lnLff^) and m = 
(lnff?(L),...,ln<T^(L)). 

To derive the second inequality, we will use the following con- 
sequence of additive majorization: Any vector a G R K majorizes 
its mean vector a whose elements are all equal to the mean; i.e., 
a ^ = Tc Hi=i & i- That is - a -< a - Now > since M = LIjH ' we 
know that Y[f=i L ii = det(LL w ) = det(M). As a result, we have 
Ylf=\ U = m det(M) and our second inequality is: 

(12) 

where h = ilndet(M). 

The proposed designs will be based on the following classes of 
functions [10]: A real-valued function /(x) defined on a subset A 
of R K is said to be Schur-convex if a -< b on A => /(a) < /(b), 
and is said to be Schur-concave if a -< b on A => /(a) > /(b). 
In particular, we will consider communication objectives that can be 
expressed as the minimization of a functions of the MSEs of each 
data stream, g(L 2 u . . . , L 2 KK ) = g(e Ll ,...,e lK ))= g{e l ). 



4.2. Schur-convex functions 

Examples of objectives that result in g(e l ) being a Schur-convex 
function of I include: minimization of the maximum of individ- 
ual MSEs: g(e l ) = max^e^; minimization of the total MMSE: 
ff( e ') ~ an d minimization of the (log) determinant MSE 

matrix: det(E) = I~[i eii > which is also Schur-concave function 
of I. For the DFE model, the SINR of the i th stream is given by 
SINRi = (1/MSEi) — 1 = e _ii - 1. Hence, many objectives in 
terms of SINR and BER can be expressed as Schur-convex functions 
of I. As we will show below, the optimal transceiver design is iden- 
tical for all these objectives. 

If g(e l ) is a Schur convex function of I, then from (12) we have 
that g(e l ) < g(e ) and the optimal value is obtained when all Zj are 
equal to U = ± lndet(M); i.e., E tl = \? tl = V det ( M )- Slnce 
the objective is an increasing function of the individual MSE, the 
design goal reduces to minimizing dot M subject to the power con- 
straint and to the constraint that diagonal elements of the Cholesky 
factor of M are all equal. We will start by characterizing the fam- 
ily of solutions that minimize dct(M) subject to the power con- 
straint, then we will show that there is a member of this fam- 
ily that yields a Cholesky factor of M with equal diagonal ele- 
ments. Minimizing dot(M) is equivalent to maximizing the Gaus- 
sian mutual information, and the family of optimal precoders is ob- 
tained using a standard water-filling algorithm [11]. In particular, if 
Rf/ = cr 2 H H R I ^ 1 H = UAhU 5 , the family of optimal precoders 



takes the form: 

P = Ui*V = Ui[* 0]V, (13) 

where Ui 6 C WtXf< ' contains the eigen vectors of Rh correspond- 
ing to the K < K largest eigen values, K and the diagonal posi- 
tive definite matrix <1? are obtained from the water- filling algorithm 
[11], and V £ C KxK is a unitary matrix degree of freedom. This 
result shows that for DFE based systems designed according to any 
Schur-convex function of I, the optimal solution is information loss- 
less. To complete the design of P, we need to select V such that 
the Cholesky decomposition of M = LL H yields an L factor with 
equal diagonal elements. Using (13): 

M = (V H (I + * T A m *r 1/2 ) ((I + * T A m *r 1/2 v) 
= LL H = R H R = (QR) H (QR), (14) 

where Ahi is the diagonal matrix containing the largest K eigen 
values of Rh, and Q is a matrix with orthonormal columns. Hence, 
finding V is equivalent to finding a V such that QR decomposition 
of (1 + # t Ahi*) _1 ^ 2 V has an R-factor with equal diagonal. This 
problem was solved in [6] and V can be obtained by applying the 
algorithm in [6] to the matrix (I + $ t Ahi*) -1 ' 2 ; see also [4, 5]. 

4.3. Schur-concave functions 

If g(e l ) is a Schur-concave function of I, then from (11) we have 
<?(e m ) < fl( e ) an d the optimal value is obtained when ha = 
<7i(L), According to Lemma 1, this equality holds when L is normal 
matrix. Since L is a lower triangular matrix, in order to be normal it 
must be a diagonal matrix [8], The optimal C in that case is I. That 
is, in the case of Schur-concave functions of I, the optimal DFE de- 
sign results in linear equalization and optimal TH precoding design 
results in linear precoding. Examples of this class of objectives in- 
clude minimization of product of the MSEs and general (weighted) 
geometrical mean of MSEs. 

5. SIMULATION STUDY 

We consider a system that transmits K = 4 streams of 16-QAM 
symbols over a 4 x 4 slowly fading independent Rayleigh channel 
with additive white Gaussian noise. We plot the average bit error rate 
(BER) against the signal to ratio P lota i/tr(R n ). We compare the per- 
formance of the proposed Schur-convex designs for THP and DFE 
(which minimize the total MSE among other objectives), with the 
corresponding linear transceiver design that minimizes total MSE 
[1, 2], and the optimal linear transceiver that maximizes the mutual 
information (minimizes logdet(E)) [1]. The performance advan- 
tages of interference cancellation are quite clear from Fig. 4. 

6. CONCLUSION 

We developed a unified framework for joint transceiver design of 
interference (pre-)subtraction schemes for MIMO channels. We ob- 
tained optimal designs for two classes of communication objectives, 
namely those that are Schur-convex and Schur-concave functions of 
the logarithms of the individual MSEs. For Schur-convex objec- 
tives, the optimal transceiver results in equal individual MSEs. For 
the DFE model, it optimizes both the total MSE and mutual infor- 
mation. For the class Schur-concave objectives, the optimal DFE 
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Fig. 4. BERs of the proposed Schur-convex designs and the optimal 
linear transceivers: minimum MSE (Linear-MMSE), and maximum 
mutual information (Linear-Det(E)), for N t = N r = K = 4. 

design results in linear equalization and the optimal TH precoding 
design results in linear precoding. 
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